Looking for Trouble and Finding It, Sometimes: Exploring Relationships between Pre-survey and Post-survey Evaluation Data

Using a new Current Population Survey (CPS) supplement on landline and cell-phone service and use as the investigative context, this case study explores relationships between presurvey evaluation data (drawn from cognitive interviews and a questionnaire appraisal coding system) and post-survey evaluation data (drawn from behavior coding and interviewer debriefings). Using qualitative data from cognitive interviews and the questionnaire appraisal system (Willis and Lessler, 1999), predictions were formulated as to where problems with the supplement questionnaire might occur during its administration in February 2004. Evidence of problems was based on behavior-coding data from 60 household interviews and on qualitative data from two focus groups conducted with CPS interviewers. Though subjective (i.e., no means of quantifying measurement error was available), the accuracy of predictions was assessed using post-survey evaluation data. A summary of predictive “hits” and “misses” is provided and discussed within the context of a larger questionnaire-design-and-evaluation framework (Esposito, 2004a, 2004b) that relates pre-survey and post-survey evaluation work.